1. Field of the Invention
The present invention relates generally to video image processing, and more particularly, to techniques for performing scaling, filtering, scan conversion, panoramic scaling, YC adjustment and color conversion on video data.
2. Background Art
Digital video is made up of many megabytes of pixel data per second of video. Processing of video data can be costly, consuming valuable resources such as memory, bandwidth, processing power, and time. Video data processing taxes even the fastest computer systems due to large image sizes and fast frame rates.
Video data consists of a series of frames or images composed of pixels arranged in rows and columns. With the advent of digital television and streaming media, a number of new techniques have been developed over the recent past to allow video images to be processed. One type of device used for video data processing is a display controller, which converts a bitstream of pixels representing an image into analog or digital signals for output on a display device. FIG. 1 illustrates the components of a conventional display controller 100.
Input video 110 represents a bitstream of pixels that is input to scan conversion unit 120. Scan conversion unit 120 converts the input video 110 to be compatible with the scan rate of a display device such as a conventional television. For example, scan conversion unit 120 may convert the input video 110 from non-interlaced frames to interlaced frames that are suitable for output at a lower scan rate. Interlaced frames generated by scan conversion unit 120 are stored in first buffer 122. Scan conversion is followed by scaling and filtering of video data.
Scaling involves changing the resolution of a frame of video data or making an image or video frame larger or smaller than its original resolution. For example, a digital video screen may be scaled up to fit a large screen, or it may be scaled down to occupy only a portion of the screen for displaying it as a thumbnail. During downscaling, the size of a frame is reduced by removing pixels. On the other hand, up scaling involves increasing the size of a frame by adding pixels. Converting a video from NTSC (640×480) resolution to HDTV (1920×1080) resolution is an example of “scaling” the video or more specifically, of up scaling. An example of downscaling would be converting from HDTV to NTSC.
When a frame is scaled, the value of each new pixel in the scaled frame is based on the values of one or more pixels in the original frame. The calculation of each new pixel can be formulated as a linear equation in which the value of each original pixel used to calculate a new pixel is multiplied by a weight or coefficient that determines the relative weight of the original pixel in the calculation. The value of a new pixel is typically calculated based its on neighboring original pixels, with different coefficients being used for different types of scaling. A frame may be scaled vertically and/or horizontally. When a frame is scaled, the ratio of the number of output pixels to the number of input pixels is referred to as the scaling ratio. For a horizontal scaling ratio of M:N, there are M horizontal pixels in the scaled frame for every N input pixels from the original frame. For example, an up scaling of 33% can be represented by a ratio of 4:3.
Filtering involves modifying the pixels in a frame based on the desired characteristics of the output frame. For example, certain filters are used to generate a blurred version of the original frame, while other filters might enhance the edges that appear in the image. Filtering is similar to scaling in that the value of each filtered pixel is calculated based on a linear equation in which the values of one or more input pixels are multiplied by coefficients.
Referring to FIG. 1, horizontal scaling unit 124 retrieves video data from first buffer 122, adjusts the size of each row in a frame and stores the resulting frame in second buffer 126. Next, horizontal filtering unit 128 retrieves the horizontally scaled data from second buffer 126, and applies a desired filter to each row of pixels. The resulting frame, which is horizontally scaled and filtered, is stored in third buffer 130. Vertical scaling unit 132 retrieves the video data from third buffer 130, performs vertical scaling to adjust the vertical size of each input frame, and stores the resulting frame in fourth buffer 134. Vertical filtering unit 136 then retrieves the vertically scaled data from fourth buffer 134, and applies a desired filter to each column of pixels to generate a vertically filtered column that is stored in fifth buffer 138.
Pixels of video data can be represented using different color spaces or color models, such as RGB and YUV. The RGB color space refers to what humans view, which is captured by a video camera or rendered on the screen of a display device. YUV is a color encoding system used for storage and processing of video data, where each pixel is represented by a luminance or brightness (Y), and two color difference signals: blue minus luminance (U) and red minus luminance (V). The Y, U, and V components of a pixel may be represented by analog signals or by digital data in the form of bits. The process of converting between RGB and YUV color spaces is referred to as color space conversion or color conversion.
Referring to FIG. 1, YC adjustment unit 140 retrieves the scan-converted, scaled and filtered video data from fifth buffer 138. YC adjustment unit 140 adjusts the contrast, brightness and saturation of a frame by adjusting the Y, U and V components of each pixel in the frame. The output of YC adjustment unit 140 is sent to color conversion unit 142, which converts video data from the YUV color space to the RGB color space for output via a display device. The output of color conversion unit 142 comprises output video 112, which is ready for viewing on the display device.
One problem with conventional video data processing is that panoramic scaling results in distortion of an image. Panoramic scaling is required when the horizontal scaling ratio for a frame is different from the vertical scaling ratio. Panoramic scaling causes distortion because objects in the scaled frame appear elongated in the horizontal or vertical direction. For example, an input frame of resolution 1280×768 pixels may need to be converted to a resolution of 1365×768, where the horizontal scaling ratio is 1365:1280, while the vertical scaling ratio is 1:1. In the present example, objects in the output frame will appear elongated in the horizontal direction because the horizontal scaling ratio is greater than the vertical scaling ratio.
Another problem with conventional video data processing is delays introduced during separate operations of scaling, filtering, scan conversion, YC adjustment and color conversion. Video processing is most useful when it can be done at actual video speeds or faster. Video processing that occurs at slower than video speeds requires large amounts of storage and cannot be viewed in real time. Conventional display controllers implement scaling, filtering and scan conversion as separate operations. Similarly, in conventional display controllers, YC adjustment and color conversion are implemented as separate operations. These separate operations require large amounts of processing power, thereby introducing delays in video processing. Further, these separate operations require numerous logic gates and large buffers, increasing both the chip size and power consumption. This is particularly problematic for video processing system-on-chip (SOC) design, where small chip size and low power dissipation are highly desirable features.
Accordingly, there is a need for a system and method that performs panoramic scaling while reducing visible distortion in the scaled frame. Further, there is a need for a system and method for efficiently performing the operations of scaling, filtering, scan conversion, YC adjustment and color conversion on video data streams. Moreover, there is a need for reducing the size and power requirements of video manipulation hardware by reducing the number of logic gates and memory buffers required for different operations.